The ability to perform effective planning is crucial for building an instructionfollowing agent.

Vision-and-Language Navigation (VLN) aims to develop embodied agents that navigate based on human instructions. However, current VLN frameworks often rely on static environments and optimal expert supervision, limiting their real-world applicability. To address this, we introduce Human-Aware Vision-and-Language Navigation (HA-VLN), extending traditional VLN by incorporating dynamic human activities and relaxing key assumptions.

Vision-and-Language Navigation requires the agent to follow language instructions to navigate through 3D environments.

Vision-and-language navigation in the real-world is an important step towards building mobile agents that perceivetheir environments and complete specific tasks following human instructions.

HAMT efficientlyencodes allthepastpanoramic observationsviaahierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history.

Table 1 presents the impacts of different random seeds for sampling the interference images. Experiments in the main manuscript are based on seed-1 which has an average performance. Figure 1: Navigation examples in normal and high-frequency perturbed scenes. In the examples shown in Figure 4, both models obtained similar textual attention. In Figure 6, according to the given instruction, the agent should turn left to enter the room corresponding to the second view.